from sklearn_benchmarks.reporting.hp_match import HPMatchReporting

scikit-learn vs. ONNX Runtime

reporting = HPMatchReporting(against_lib="onnx", config="config.yml")
reporting.make_report()

We assume here there is a perfect match between the hyperparameters of both librairies. For a given set of parameters and a given dataset, we compute the speedup time scikit-learn / time onnx. For instance, a speedup of 2 means that onnx is twice as fast as scikit-learn for a given set of parameters and a given dataset.

KNeighborsClassifier_brute_force

onnx (1.10.1) vs. scikit-learn (1.0.dev0)

All estimators share the following parameters: algorithm=brute.

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	n_iter	iteration_throughput	latency	n_jobs	n_neighbors	accuracy_score	mean_duration_onnx	std_duration_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
0	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.011	0.000	NaN	7.192	0.0	-1	1	NaN	18.236	0.208	0.001	0.001	See	See
3	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.011	0.000	NaN	7.250	0.0	-1	5	NaN	0.327	0.012	0.034	0.034	See	See
6	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.012	0.001	NaN	6.850	0.0	1	100	NaN	18.368	0.027	0.001	0.001	See	See
9	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.012	0.000	NaN	6.917	0.0	-1	100	NaN	0.342	0.014	0.034	0.034	See	See
12	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.011	0.000	NaN	6.992	0.0	1	5	NaN	3.828	0.068	0.003	0.003	See	See
15	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.012	0.000	NaN	6.931	0.0	1	1	NaN	0.251	0.012	0.046	0.046	See	See
18	KNeighborsClassifier_brute_force	fit	100000	100000	2	0.004	0.000	NaN	0.357	0.0	-1	1	NaN	3.824	0.092	0.001	0.001	See	See
21	KNeighborsClassifier_brute_force	fit	100000	100000	2	0.005	0.000	NaN	0.336	0.0	-1	5	NaN	0.254	0.011	0.019	0.019	See	See

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	n_iter	iteration_throughput	latency	n_jobs	n_neighbors	accuracy_score	mean_duration_onnx	std_duration_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
1	KNeighborsClassifier_brute_force	predict	100000	1000	100	1.888	0.094	NaN	0.000	0.002	-1	1	0.663	0.330	0.013	5.714	5.718	See	See
2	KNeighborsClassifier_brute_force	predict	100000	1	100	0.022	0.002	NaN	0.000	0.022	-1	1	1.000	18.005	0.195	0.001	0.001	See	See
4	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.782	0.048	NaN	0.000	0.003	-1	5	0.757	18.006	0.145	0.155	0.155	See	See
5	KNeighborsClassifier_brute_force	predict	100000	1	100	0.024	0.003	NaN	0.000	0.024	-1	5	1.000	0.320	0.007	0.073	0.073	See	See
7	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.107	0.028	NaN	0.000	0.002	1	100	0.882	0.339	0.012	6.212	6.216	See	See
8	KNeighborsClassifier_brute_force	predict	100000	1	100	0.021	0.000	NaN	0.000	0.021	1	100	1.000	17.977	0.100	0.001	0.001	See	See
10	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.857	0.037	NaN	0.000	0.003	-1	100	0.882	17.779	0.080	0.161	0.161	See	See
11	KNeighborsClassifier_brute_force	predict	100000	1	100	0.024	0.002	NaN	0.000	0.024	-1	100	1.000	0.333	0.008	0.072	0.072	See	See
13	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.121	0.049	NaN	0.000	0.002	1	5	0.757	0.247	0.010	8.580	8.587	See	See
14	KNeighborsClassifier_brute_force	predict	100000	1	100	0.021	0.000	NaN	0.000	0.021	1	5	1.000	3.786	0.072	0.006	0.006	See	See
16	KNeighborsClassifier_brute_force	predict	100000	1000	100	1.156	0.013	NaN	0.001	0.001	1	1	0.663	3.829	0.132	0.302	0.302	See	See
17	KNeighborsClassifier_brute_force	predict	100000	1	100	0.019	0.000	NaN	0.000	0.019	1	1	1.000	0.261	0.006	0.073	0.073	See	See
19	KNeighborsClassifier_brute_force	predict	100000	1000	2	1.722	0.026	NaN	0.000	0.002	-1	1	0.896	0.253	0.009	6.792	6.797	See	See
20	KNeighborsClassifier_brute_force	predict	100000	1	2	0.006	0.004	NaN	0.000	0.006	-1	1	1.000	3.790	0.077	0.002	0.002	See	See
22	KNeighborsClassifier_brute_force	predict	100000	1000	2	2.633	0.032	NaN	0.000	0.003	-1	5	0.922	3.850	0.053	0.684	0.684	See	See
23	KNeighborsClassifier_brute_force	predict	100000	1	2	0.006	0.002	NaN	0.000	0.006	-1	5	1.000	0.253	0.011	0.024	0.024	See	See

KNeighborsClassifier_kd_tree

onnx (1.10.1) vs. scikit-learn (1.0.dev0)

All estimators share the following parameters: algorithm=kd_tree.

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	n_iter	iteration_throughput	latency	n_jobs	n_neighbors	accuracy_score	mean_duration_onnx	std_duration_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
0	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	2.885	0.091	NaN	0.028	0.0	-1	1	NaN	120.745	0.000	0.024	0.024	See	See
3	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	3.484	0.055	NaN	0.023	0.0	-1	5	NaN	2.747	0.234	1.268	1.273	See	See
6	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	3.486	0.043	NaN	0.023	0.0	1	100	NaN	121.589	0.000	0.029	0.029	See	See
9	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	3.515	0.098	NaN	0.023	0.0	-1	100	NaN	2.817	0.184	1.248	1.250	See	See
12	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	3.447	0.071	NaN	0.023	0.0	1	5	NaN	0.053	0.030	64.661	74.431	See	See
15	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	3.508	0.065	NaN	0.023	0.0	1	1	NaN	0.006	0.000	600.773	601.211	See	See
18	KNeighborsClassifier_kd_tree	fit	1000	1000	2	0.001	0.001	NaN	0.019	0.0	-1	1	NaN	0.062	0.003	0.014	0.014	See	See
21	KNeighborsClassifier_kd_tree	fit	1000	1000	2	0.001	0.000	NaN	0.030	0.0	-1	5	NaN	0.005	0.000	0.099	0.099	See	See

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	n_iter	iteration_throughput	latency	n_jobs	n_neighbors	accuracy_score	mean_duration_onnx	std_duration_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
1	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	0.826	1.023	NaN	0.000	0.001	-1	1	0.929	2.734	0.230	0.302	0.303	See	See
2	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.003	0.001	NaN	0.000	0.003	-1	1	1.000	121.543	0.000	0.000	0.000	See	See
4	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	1.000	0.348	NaN	0.000	0.001	-1	5	0.946	119.787	0.000	0.008	0.008	See	See
5	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.003	0.001	NaN	0.000	0.003	-1	5	1.000	2.799	0.211	0.001	0.001	See	See
7	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	5.175	0.523	NaN	0.000	0.005	1	100	0.951	2.763	0.238	1.873	1.880	See	See
8	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.003	0.001	NaN	0.000	0.003	1	100	1.000	120.530	0.000	0.000	0.000	See	See
10	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	2.791	0.325	NaN	0.000	0.003	-1	100	0.951	120.106	0.000	0.023	0.023	See	See
11	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.005	0.001	NaN	0.000	0.005	-1	100	1.000	2.730	0.199	0.002	0.002	See	See
13	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	1.601	0.394	NaN	0.000	0.002	1	5	0.946	0.006	0.001	284.508	286.132	See	See
14	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.002	0.000	NaN	0.000	0.002	1	5	1.000	0.045	0.001	0.036	0.036	See	See
16	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	0.909	0.364	NaN	0.000	0.001	1	1	0.929	0.063	0.007	14.469	14.556	See	See
17	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.001	0.000	NaN	0.000	0.001	1	1	1.000	0.006	0.000	0.180	0.180	See	See
19	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.026	0.016	NaN	0.001	0.000	-1	1	0.891	0.005	0.000	4.970	4.982	See	See
20	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.002	0.000	NaN	0.000	0.002	-1	1	1.000	0.039	0.002	0.053	0.053	See	See
22	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.024	0.002	NaN	0.001	0.000	-1	5	0.911	0.039	0.002	0.631	0.631	See	See
23	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.002	0.001	NaN	0.000	0.002	-1	5	1.000	0.005	0.000	0.437	0.438	See	See

HistGradientBoostingClassifier_best

onnx (1.10.1) vs. scikit-learn (1.0.dev0)

All estimators share the following parameters: learning_rate=0.01, n_iter_no_change=10.0, max_leaf_nodes=100.0, max_bins=255.0, min_samples_leaf=100.0, max_iter=300.0.

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	n_iter	iteration_throughput	latency	accuracy_score	mean_duration_onnx	std_duration_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
0	HistGradientBoostingClassifier_best	fit	100000	100000	100	98.321	0.0	300	0.001	0.001	NaN	0.495	0.021	198.584	198.763	See	See

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	n_iter	iteration_throughput	latency	accuracy_score	mean_duration_onnx	std_duration_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
1	HistGradientBoostingClassifier_best	predict	100000	1000	100	0.116	0.006	300	0.007	0.0	0.824	0.417	0.016	0.279	0.279	See	See